By Doc Searls
I always wear a Linux Journal shirt at conferences. One reason, of course, is company loyalty. Another is the fact that we put out some fine shirts. But the best reason of all is that it attracts stories.
People love to tell their Linux stories. The best are suitable for publication.
Which is what happened at the Personalization Summit in New York (April 2-4), where I was on a panel about "The Future of Personalization" (briefly, I said there wasn't much hope). Sitting near me in the audience was John K. Thompson, VP Worldwide Marketing for a company called WhiteCross.
"You know," he said, "we believe we have the world's fastest Linux analyticical platform."
A couple hours later we met again, and I decided to interview him. Here's the result:
DS: Where did the company come from?
JKT: We were founded in '92 by guys who left Terradata. From the beginning we were in the business intelligence business: crunching huge amounts ofnalytical data. So we built a platform that was about analytical speed: all-out speed. So we started by building a massively parallel computational system based on Lynx (Now LynuxWorks). The hardeare platform was built from pure commodity products: AMD 333MHz chips, IBM SSA drives, Ethernet interconnects.
Then we started to work on the next generation, called lightning. It will go up to 1.2 GHz chips, bigger drives; but still rack-mounted, and still 50 times the performace of the NCR and Sun StarFire boxes.
DS: And this is where Linux came in?
JKT: Right. We decided what we would do on the new platform is port the existing platform over to Linux. We now run our ASP in our datacenters in San Francisco and Bracknell, outside of London, and sell services on top of those. We offer turnkey outsourced turnkey and analytical solutions.
The way this is written is all about analytics. SQL interface via ODBC. The data exploration server is a two tier architected system. The first layer is a Sun UNIX platform running Solaris. These are our communicationss processors, which interface to external systems and take in data feeds: loads, streams. We can run these in parallel. Whether ASCII, EBCDIC or whatever, we convert the data into binary and load it into the massively parallel part of the system: the DES (Data Exploration Server). This is what used to run on Lynx and now runs on Linux. We can set these up so they're always running in parallel.
DS: What are the performance improvements over the old system?
JKT: At Freeserve, a large ISP in the UK, it took 28 hours to do their log files to do daily reporting. We came in and set up the load process on a one-rack system. It loaded in 50 minutes.
We have a table where we compared the new and the old OSes. About 105 operations. When we averaged them all out, Linux came out 20 times faster. What we do is all about performance. We wouldn't have made the change if the difference wasn't so high. Betwen 85 and 95 were faster on Linux. Access to mem, dcisk reads, swapping cache, disk writes... Linux kicked butt.
DS: Tell me more about how this works.
JKT: Okay, we have this two-tiered system that is massively parallel and can grow with any client requirements. One of the technical benefits is that a database made of homogenous data is just as easy to maintain and operate as one made of fifty different data sources. We're taking the entire database, bringing it to binary and analysing the entire base every time we run against it. So rather than optimization, indexes and other performance enhancing techniques that DBAs (database administrators) use, we just put in the data and run against all of it all of the time. Just snap in more disk and processor cards. Grow incrementally. The TCO (total cost of operation) is marginal compared to a Sun or an Oracle implementation. We're not worried about user queries, usage patterns or load times, which are where DBAs spend most of their time. Their question is "how do you load two terrabytes of data in the batch window?' We have an implementation in the U.S. where we're loading 85 gigabytes a day in real time, and the system isn't even straining.
From a customer perspective, people want to look at divergent sets of data: product, customer, usage, whatever. And what we've traditionally done is made them build different databases. That fragments the view of the organization and the customers. While we're loading so much data per day we're taking a census of the activity of the customer, the services, the products they're using, and with this unified date ensemble. We're enableing them to look at all the views of the business in one data set. You can look at customers, products, services, network capacity, pricing analysis, strategic planning on market entry — all on one data set.
DS: Is this a standalone system?
JKT: This goes in as a complement to other platforms. It's adds scaling of data and analytical processing. It doesn't displace the data warehouse. We're talking about building a facility that allows simple reporting, multidimensional analysis, data mining and data exploration in a very active platform. If you look at most data warehouses, they're pretty static. We're offering them a dynamic facility. This dynamism is allowed — at virtually no cost whatsoever — by Linux.
DS: So why Linux?
JKT: It's so flexible. People usually hear about Linux as a file or a print or a Web server. We're asking it to do a similar thing. Swapping memory, moving data on and off disk, parsing an SQL statement, and a ton of computation. The OS does everything in the box. But the key fact that Linux makes it possible to plug and crunch. Just add more racks.
And it's easy to monitor and administer. We monitor client systems constantly, sending alerts saying you 're going to go critical on disk, so they can give us the signal to add more cards. Then it's just a hardware cost. The system reconfigures itself and away it goes. It senses the presence of another card, decides if it's a processor or disk card, addresses it and carries on.
A lot of peoplle think there's nothing new in hardware, but this is a new and simple way to add functionality by adding components that work together in a way nobody thought they would.
DS: I imagine cost is also a benefit.
JKT: Right. Go get a hot dog and some Linux. But the openness is a huge benefit too. What we're doing is an open systms approach engineered in a unique way. People accept Sun's Starfire and NCR's Worldmark servers as open, but it's not true. Linux is truly open. Its so eady. When you've commodity chips strapped together with commodity drives hooked together with fast ethernet interconnects, then you want a commmodiy OS, and Linux is it.
DS: Are you using a Linux distribution, or something from one of the embedded tool chain companies?
JKT: The Linux is all ours. We took the basic kernel and extended it with our own drivers, just like we had done with Lynx.
DS: Where did the company come from?